Multifusion of a stream operator in a streaming application

ABSTRACT

Embodiments of the present disclosure include a method, a system, and a computer program product for fusing a stream operator into more than one processing element within a streaming application. The method includes receiving an instruction to concurrently fuse, into a second processing element, a stream operator of interest that is fused into a first processing element. The method includes determining whether the stream operator of interest is stateful. The method includes compiling, in response to determining the stream operator of interest is stateful, a clone of the stream operator of interest into the second processing element so that the clone is synchronized with the stream operator of interest.

BACKGROUND

The present disclosure relates to stream computing, and in particular,to computing applications that receive streaming data and process thedata as it is received.

Database systems are typically configured to separate the process ofstoring data from accessing, manipulating, or using data stored in adatabase. More specifically, database systems use a model in which datais first stored and indexed in a memory before subsequent querying andanalysis. In general, database systems may not be well suited forperforming real-time processing and analyzing streaming data. Inparticular, database systems may be unable to store, index, and analyzelarge amounts of streaming data efficiently or in real time.

SUMMARY

Embodiments of the present disclosure include a method, a system, and acomputer program product for fusing a stream operator into more than oneprocessing element within a streaming application.

One embodiment is directed toward a method for fusing a stream operatorinto more than one processing element within a streaming application.The method includes receiving an instruction to concurrently fuse, intoa second processing element, a stream operator of interest that is fusedinto a first processing element. The method includes determining whetherthe stream operator of interest is stateful. The method includescompiling, in response to determining the stream operator of interest isstateful, a clone of the stream operator of interest into the secondprocessing element so that the clone is synchronized with the streamoperator of interest.

Another embodiment is directed toward a system for fusing a streamoperator into more than one processing element within a streamingapplication. The system includes a compute node. The compute node hostsa first processing element and a second processing element. A streamoperator of interest is fused into the first processing element. Thesystem also includes a fusion manager that is configured to receive aninstruction to concurrently fuse, into the second processing element,the stream operator of interest. The fusion manager is configured todetermine whether the stream operator of interest is stateful. Thefusion manager is configured to provide a clone instruction, in responseto determining the stream operator of interest is stateful. The systemalso includes a compiler. The compiler is configured to compile, basedon the clone instruction, a clone of the stream operator of interestinto the second processing element. The clone is synchronized with thestream operator of interest.

Yet another embodiment is directed toward a computer program product.

The above summary is not intended to describe each illustratedembodiment or every implementation of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings included in the present application are incorporated into,and form part of, the specification. They illustrate embodiments of thepresent disclosure and, along with the description, serve to explain theprinciples of the disclosure. The drawings are only illustrative ofcertain embodiments and do not limit the disclosure.

FIG. 1 illustrates a computing infrastructure configured to execute astream computing application according to various embodiments.

FIG. 2 illustrates a more detailed view of a compute node of FIG. 1according to various embodiments.

FIG. 3 illustrates a more detailed view of the management system of FIG.1 according to various embodiments.

FIG. 4 illustrates a more detailed view of the compiler system of FIG. 1according to various embodiments.

FIG. 5 illustrates an operator graph for a stream computing applicationaccording to various embodiments.

FIG. 6 illustrates a flowchart of a method for fusing a stream operatorto more than one processing element, according to various embodiments.

FIG. 7 illustrates a flowchart of a method of cloning a stream operatorof interest, according to various embodiments.

FIG. 8 illustrates a flowchart of a method for determining whether tounfuse the stream operator of interest or clone from the secondprocessing element.

FIG. 9A illustrates a stateless embodiment of an operator graph in amulti-fusion configuration, according to various embodiments.

FIG. 9B illustrates a stateful embodiment of an operator graph in amulti-fusion configuration, according to various embodiments.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

The present disclosure relates to stream computing, and in particular,to computing applications that receive streaming data and process thedata as it is received. For instance, a processing element canordinarily be configured to contain a stream operator of interest froman operator graph. The stream operator of interest receives the data andprocesses the data as it is received. One or more stream operators arefused to one processing element. In a multi-fusion operation, the streamoperator of interest is fused into a plurality of processing elementsand the computing performance of a stream operator of interest can beimproved.

A stream operator of interest can be fused by more than one processingelement by creating a clone of the stream operator of interest andfusing the clone within another processing element. Aspects of thepresent disclosure also provide for a synchronization mechanism betweenthe stream operator of interest and the clone. The clone can be a copyof the stream operator of interest that inherits the variousdependencies and methods of the stream operator of interest. While thepresent disclosure is not necessarily limited to such applications,various aspects of the disclosure may be appreciated through adiscussion of various examples using this context.

A virtual resource refers to resources that are distributed to virtualmachines. Virtual resources can include computational resources such asCPU access, memory, network bandwidth, storage but can also includeexport regulations and encryption. For example, if an export regulationis used, then the geographic location of a server can be a virtualresource. The number of virtual machines that be hosted by servers basedin a particular country may be restricted.

Although not necessarily limited thereto, embodiments of the presentdisclosure can be appreciated in the context of streaming data andproblems relating to cloning stream operators into separate processingelements. Throughout this disclosure, the term template elements cangenerically refer to a processing element or a stream operator used in atemplate. Abbreviations used can include “S.O.” or “OP” for streamoperator, and P.E./PE for processing element, and VM for virtualmachine.

Stream-based computing and stream-based database computing are emergingas a developing technology for database systems. Products are availablewhich allow users to create applications that process and querystreaming data before it reaches a database file. With this emergingtechnology, users can specify processing logic to apply to inbound datarecords while they are “in flight,” with the results available in a veryshort amount of time, often in fractions of a second. Constructing anapplication using this type of processing has opened up a newprogramming paradigm that will allow for development of a broad varietyof innovative applications, systems, and processes, as well as presentnew challenges for application programmers and database developers.

In a stream computing application, stream operators are connected to oneanother such that data flows from one stream operator to the next (e.g.,over a TCP/IP socket). When a stream operator receives data, it mayperform operations, such as analysis logic, which may change the tupleby adding or subtracting attributes, or updating the values of existingattributes within the tuple. When the analysis logic is complete, a newtuple is then sent to the next stream operator. Scalability is achievedby distributing an application across nodes by creating executables(i.e., processing elements), as well as replicating processing elementson multiple nodes and load balancing among them. Stream operators in astream computing application can be fused together to form a processingelement that is executable. Doing so allows processing elements to sharea common process space, resulting in much faster communication betweenstream operators than is available using inter-process communicationtechniques (e.g., using a TCP/IP socket). Further, processing elementscan be inserted or removed dynamically from an operator graphrepresenting the flow of data through the stream computing application.A particular stream operator may not reside within the same operatingsystem process as other stream operators. In addition, stream operatorsin the same operator graph may be hosted on different nodes, e.g., ondifferent compute nodes or on different cores of a compute node.

Data flows from one stream operator to another in the form of a “tuple.”A tuple is a sequence of one or more attributes associated with anentity. Attributes may be any of a variety of different types, e.g.,integer, float, Boolean, string, etc. The attributes may be ordered. Inaddition to attributes associated with an entity, a tuple may includemetadata, i.e., data about the tuple. A tuple may be extended by addingone or more additional attributes or metadata to it. As used herein,“stream” or “data stream” refers to a sequence of tuples. Generally, astream may be considered a pseudo-infinite sequence of tuples.

Nonetheless, an output tuple may be changed in some way by a streamoperator or processing element. An attribute or metadata may be added,deleted, or modified. For example, a tuple will often have two or moreattributes. A stream operator or processing element may receive thetuple having multiple attributes and output a tuple corresponding withthe input tuple. The stream operator or processing element may onlychange one of the attributes so that all of the attributes of the outputtuple except one are the same as the attributes of the input tuple.

Generally, a particular tuple output by a stream operator or processingelement may not be considered to be the same tuple as a correspondinginput tuple even if the input tuple is not changed by the processingelement. However, to simplify the present description and the claims, anoutput tuple that has the same data attributes or is associated with thesame entity as a corresponding input tuple will be referred to herein asthe same tuple unless the context or an express statement indicatesotherwise.

Stream computing applications handle massive volumes of data that needto be processed efficiently and in real time. For example, a streamcomputing application may continuously ingest and analyze hundreds ofthousands of messages per second and up to petabytes of data per day.Accordingly, each stream operator in a stream computing application maybe required to process a received tuple within fractions of a second.Unless the stream operators are located in the same processing element,it is necessary to use an inter-process communication path each time atuple is sent from one stream operator to another. Inter-processcommunication paths can be a critical resource in a stream computingapplication. According to various embodiments, the available bandwidthon one or more inter-process communication paths may be conserved.Efficient use of inter-process communication bandwidth can speed upprocessing.

An operator graph can be an execution path for a plurality of streamoperators to process a stream of tuples. In addition to streamoperators, the operator graph can refer to an execution path forprocessing elements and the dependent stream operators of the processingelements to process the stream of tuples. Generally, the operator graphcan contain a plurality of stream operators that produce a particularend result, e.g., calculate an average.

FIG. 1 illustrates one exemplary computing infrastructure 100 that maybe configured to execute a stream computing application, according tosome embodiments. The computing infrastructure 100 includes a managementsystem 105 and two or more compute nodes 110A—110D—e.g., hosts or aresource/partition in a cloud computing environment—which arecommunicatively coupled to each other using one or more communicationsnetworks 120. The communications network 120 may include one or moreservers, networks, or databases, and may use a particular communicationprotocol to transfer data between the compute nodes 110A—110D.

A compiler system 102 may be communicatively coupled with the managementsystem 105 and the compute nodes 110 either directly or via thecommunications network 120. The compiler system 102 can further includea compiler 136. The compiler 136 is responsible for forming thecommunication links between the processing elements and between thestream operators. The compute nodes 110A-110D can be a hardware resourcethat supports the operation of the processing of the stream of tuples.The compute nodes 110A-110D can also be the hardware resources for acloud computing environment. As discussed herein, the compute nodes110A-110D can also be the virtualization platform for virtual machines.

The management system 105 can control the management of the computenodes 110A-110D (discussed further on FIG. 3). In various embodiments,the management system 105 is a compute node configured to be runningstream management software. The management system 105 can have anoperator graph 132 with one or more stream operators and a streammanager 134 to control the management of the stream of tuples in theoperator graph 132. The stream manager 134 can manage the processes fromthe operator graph, including anything associated with the operatorgraph 132. The stream manager 134 can include a stream operator monitor140 that monitors the stream operators in the operator graph forparticular metrics. The management system 105 can also include a fusionmanager 145 that communicates instructions to the compiler 136. Forexample, the fusion manager 145 can receive a performance metric for astream operator from the stream manager 134 and make the determinationon whether to recompile the operator graph 132. In various embodiments,the fusion manager 145 can communicate instructions to modify the fusingof processing elements with the compiler 136. The fusion manager 145 canbe located in the stream manager 134 or the management system 105.

The communications network 120 may include a variety of types ofphysical communication channels or “links.” The links may be wired,wireless, optical, or any other suitable media. In addition, thecommunications network 120 may include a variety of network hardware andsoftware for performing routing, switching, and other functions, such asrouters, switches, or bridges. The communications network 120 may bededicated for use by a stream computing application or shared with otherapplications and users. The communications network 120 may be any size.For example, the communications network 120 may include a single localarea network or a wide area network spanning a large geographical area,such as the Internet. The links may provide different levels ofbandwidth or capacity to transfer data at a particular rate. Thebandwidth that a particular link provides may vary depending on avariety of factors, including the type of communication media andwhether particular network hardware or software is functioning correctlyor at full capacity. In addition, the bandwidth that a particular linkprovides to a stream computing application may vary if the link isshared with other applications and users. The available bandwidth mayvary depending on the load placed on the link by the other applicationsand users. The bandwidth that a particular link provides may also varydepending on a temporal factor, such as time of day, day of week, day ofmonth, or season.

FIG. 2 is a more detailed view of a compute node 110, which may be thesame as one of the compute nodes 110A-110D of FIG. 1, according tovarious embodiments. The compute node 110 may include, withoutlimitation, one or more processors (CPUs) 205, a network interface 215,an interconnect 220, a memory 225, and a storage 230. The compute node110 may also include an I/O device interface 210 used to connect I/Odevices 212, e.g., keyboard, display, and mouse devices, to the computenode 110.

Each CPU 205 retrieves and executes programming instructions stored inthe memory 225 or storage 230. Similarly, the CPU 205 stores andretrieves application data residing in the memory 225. The interconnect220 is used to transmit programming instructions and application databetween each CPU 205, I/O device interface 210, storage 230, networkinterface 215, and memory 225. The interconnect 220 may be one or morebusses. The CPUs 205 may be a single CPU, multiple CPUs, or a single CPUhaving multiple processing cores in various embodiments. In oneembodiment, a processor 205 may be a digital signal processor (DSP). Oneor more processing elements 235 (described below) may be stored in thememory 225. A processing element 235 may include one or more streamoperators 240 (described below). In one embodiment, a processing element235 is assigned to be executed by only one CPU 205, although in otherembodiments the stream operators 240 of a processing element 235 mayinclude one or more threads that are executed on two or more CPUs 205.The memory 225 is generally included to be representative of a randomaccess memory, e.g., Static Random Access Memory (SRAM), Dynamic RandomAccess Memory (DRAM), or Flash. The storage 230 is generally included tobe representative of a non-volatile memory, such as a hard disk drive,solid state device (SSD), or removable memory cards, optical storage,flash memory devices, network attached storage (NAS), or connections tostorage area network (SAN) devices, or other devices that may storenon-volatile data. The network interface 215 is configured to transmitdata via the communications network 120.

A compute node 110 can be configured to have a hypervisor 245. Thehypervisor 245 can be configured to distribute the hardware elements,e.g., the CPUs 205, the memory 225, the storage 230, to the virtualmachines 250. The hypervisor 245 can run as a separate program or beimbedded into the firmware of the compute node 110. The virtual machine250 can replicate the experience of a standalone compute node to avariety of users without degrading performance. Due to spikes in demand,the hypervisor 245 can be configured to dynamically distribute thehardware resources to the virtual machine where the hardware resourcesof the compute node are most demanded.

A stream computing application may include one or more stream operators240 that may be compiled into a “processing element” container 235. Twoor more processing elements 235 may run on the same memory 225, eachprocessing element having one or more stream operators 240. Each streamoperator 240 may include a portion of code that processes tuples flowinginto a processing element and outputs tuples to other stream operators240 in the same processing element, in other processing elements, or inboth the same and other processing elements in a stream computingapplication. Processing elements 235 may pass tuples to other processingelements that are on the same compute node 110 or on other compute nodesthat are accessible via communications network 120. For example, aprocessing element 235 on compute node 110A may output tuples to aprocessing element 235 on compute node 110B.

The storage 230 may include a buffer 260. Although shown as being instorage, the buffer 260 may be located in the memory 225 of the computenode 110 or in a combination of both memories. Moreover, storage 230 mayinclude storage space that is external to the compute node 110, such asin a cloud.

The compute node 110 may include one or more operating systems 262. Anoperating system 262 may be stored partially in memory 225 and partiallyin storage 230. Alternatively, an operating system may be storedentirely in memory 225 or entirely in storage 230. The operating systemprovides an interface between various hardware resources, including theCPU 205, and processing elements and other components of the streamcomputing application. In addition, an operating system provides commonservices for application programs, such as providing a time function.

FIG. 3 is a more detailed view of the management system 105 of FIG. 1according to some embodiments. The management system 105 may include,without limitation, one or more processors (CPUs) 305, a networkinterface 315, an interconnect 320, a memory 325, and a storage 330. Themanagement system 105 may also include an I/O device interface 310connecting I/O devices 312, e.g., keyboard, display, and mouse devices,to the management system 105.

Each CPU 305 retrieves and executes programming instructions stored inthe memory 325 or storage 330. Similarly, each CPU 305 stores andretrieves application data residing in the memory 325 or storage 330.The interconnect 320 is used to move data, such as programminginstructions and application data, between the CPU 305, I/O deviceinterface 310, storage unit 330, network interface 315, and memory 325.The interconnect 320 may be one or more busses. The CPUs 305 may be asingle CPU, multiple CPUs, or a single CPU having multiple processingcores in various embodiments. In one embodiment, a processor 305 may bea DSP. Memory 325 is generally included to be representative of a randomaccess memory, e.g., SRAM, DRAM, or Flash. The storage 330 is generallyincluded to be representative of a non-volatile memory, such as a harddisk drive, solid state device (SSD), removable memory cards, opticalstorage, Flash memory devices, network attached storage (NAS),connections to storage area-network (SAN) devices, or the cloud. Thenetwork interface 315 is configured to transmit data via thecommunications network 120.

The memory 325 may store a stream manager 134. The stream manager 134can have software features that manage the stream of tuples throughoperator graph 335. The stream manager 134 can also perform aspects ofthe disclosure. For example, the stream manager 134 can modify theoperator graph 335 to increase or decrease threads between streamoperators. The stream manager 134 can include a stream operator monitor140 to monitor the performance of a stream operator and collect metricswithin the operator graph 335. The memory 325 may also store a fusionmanager 145. The fusion manager 145 can optionally be part of the streammanager 134. The fusion manager 145 communicates with the stream manager134 and the compiler to determine whether the stream operator ofinterest can have a multi-fusion operation performed.

According to various embodiments, an operator graph 335 can run inmemory 325 and the corresponding data tuples 340 can be processed andstored in databases associated with the storage element 330. The streammanager 134 can also require a database when reading or writing fromstorage 330 and logging from storage 330.

Additionally, the storage 330 may store an operator graph 335. Theoperator graph 335 may define how tuples are routed to processingelements 235 (FIG. 2) for processing.

The management system 105 may include one or more operating systems 332.An operating system 332 may be stored partially in memory 325 andpartially in storage 330. Alternatively, an operating system may bestored entirely in memory 325 or entirely in storage 330. The operatingsystem provides an interface between various hardware resources,including the CPU 305, and processing elements and other components ofthe stream computing application. In addition, an operating systemprovides common services for application programs, such as providing atime function.

FIG. 4 is a more detailed view of the compiler system 102 of FIG. 1according to some embodiments. The compiler system 102 may include,without limitation, one or more processors (CPUs) 405, a networkinterface 415, an interconnect 420, a memory 425, and storage 430. Thecompiler system 102 may also include an I/O device interface 410connecting I/O devices 412, e.g., keyboard, display, and mouse devices,to the compiler system 102.

Each CPU 405 retrieves and executes programming instructions stored inthe memory 425 or storage 430. Similarly, each CPU 405 stores andretrieves application data residing in the memory 425 or storage 430.The interconnect 420 is used to move data, such as programminginstructions and application data, between the CPU 405, I/O deviceinterface 410, storage unit 430, network interface 415, and memory 425.The interconnect 420 may be one or more busses. The CPUs 405 may be asingle CPU, multiple CPUs, or a single CPU having multiple processingcores in various embodiments. In one embodiment, a processor 405 may bea DSP. Memory 425 is generally included to be representative of a randomaccess memory, e.g., SRAM, DRAM, or Flash. The storage 430 is generallyincluded to be representative of a non-volatile memory, such as a harddisk drive, solid state device (SSD), removable memory cards, opticalstorage, flash memory devices, network attached storage (NAS),connections to storage area-network (SAN) devices, or to the cloud. Thenetwork interface 415 is configured to transmit data via thecommunications network 120.

The compiler system 102 may include one or more operating systems 432.An operating system 432 may be stored partially in memory 425 andpartially in storage 430. Alternatively, an operating system may bestored entirely in memory 425 or entirely in storage 430. The operatingsystem provides an interface between various hardware resources,including the CPU 405, and processing elements and other components ofthe stream computing application. In addition, an operating systemprovides common services for application programs, such as providing atime function.

The memory 425 may store a compiler 136. The compiler 136 compilesmodules, which include source code or statements, into the object code,which includes machine instructions that execute on a processor. In oneembodiment, the compiler 136 may translate the modules into anintermediate form before translating the intermediate form into objectcode. The compiler 136 may output a set of deployable artifacts that mayinclude a set of processing elements and an application descriptionlanguage file (ADL file), which is a configuration file that describesthe stream computing application. In some embodiments, the compiler 136may be a just-in-time compiler that executes as part of an interpreter.In other embodiments, the compiler 136 may be an optimizing compiler. Invarious embodiments, the compiler 136 may perform peepholeoptimizations, local optimizations, loop optimizations, inter-proceduralor whole-program optimizations, machine code optimizations, or any otheroptimizations that reduce the amount of time required to execute theobject code, to reduce the amount of memory required to execute theobject code, or both. The output of the compiler 136 may be representedby an operator graph, e.g., the operator graph 335.

The compiler 136 may also communicate with a fusion manager from themanagement system to provide the application administrator with theability to optimize performance through profile-driven fusionoptimization. Any decision to fuse operators requires balancing thebenefits of distributing processing across multiple compute nodes withthe benefit of faster inter-operator communications. The compiler 136may automate the fusion process through the fusion manager in themanagement system to determine how to best fuse the operators to behosted by one or more processing elements, while respectinguser-specified constraints. This may be a two-step process, includingcompiling the application in a profiling mode and running theapplication, then re-compiling and using the optimizer during thissubsequent compilation. The end result may, however, be acompiler-supplied deployable application with an optimized applicationconfiguration.

FIG. 5 illustrates an exemplary operator graph 500 for a streamcomputing application beginning from one or more sources 135 through toone or more sinks 504, 506, according to some embodiments. This flowfrom source to sink may also be generally referred to herein as anexecution path. In addition, a flow from one processing element toanother may be referred to as an execution path in various contexts. Theoperator graph 500 may be comparable to the operator graph 132 inFIG. 1. In various embodiments, the operator graph 132 in FIG. 1represents the management of the operator graph 500. Although FIG. 5 isabstracted to show connected processing elements PE1-PE10, the operatorgraph 500 may include data flows between stream operators 240 (FIG. 2)within the same or different processing elements. Typically, processingelements, such as processing element 235 (FIG. 2), receive tuples fromthe stream as well as output tuples into the stream (except for asink—where the stream terminates, or a source—where the stream begins).While the operator graph 500 includes a relatively small number ofcomponents, an operator graph may be much more complex and may includemany individual operator graphs that may be statically or dynamicallylinked together.

The example operator graph shown in FIG. 5 includes ten processingelements (labeled as PE1-PE10) running on the compute nodes 110A-110D. Aprocessing element may include one or more stream operators fusedtogether to form an independently running process with its own processID (PID) and memory space. In cases where two (or more) processingelements are running independently, inter-process communication mayoccur using a “transport,” e.g., a network socket, a TCP/IP socket, orshared memory. Inter-process communication paths used for inter-processcommunications can be a critical resource in a stream computingapplication. However, when stream operators are fused together, thefused stream operators can use more rapid communication techniques forpassing tuples among stream operators in each processing element.

The operator graph 500 begins at a source 135 and ends at a sink 504,506. Compute node 110A includes the processing elements PE1, PE2, andPE3. Source 135 flows into the processing element PE1, which in turnoutputs tuples that are received by PE2 and PE3. For example, PE1 maysplit data attributes received in a tuple and pass some data attributesin a new tuple to PE2, while passing other data attributes in anothernew tuple to PE3. As a second example, PE1 may pass some received tuplesto PE2 while passing other tuples to PE3. Tuples that flow to PE2 areprocessed by the stream operators contained in PE2, and the resultingtuples are then output to PE4 on compute node 110B Likewise, the tuplesoutput by PE4 flow to PE6 and sink 504. Similarly, tuples flowing fromPE3 to PE5 also reach the operators in sink PE6 504. Thus, in additionto being a sink for this example operator graph, PE6 could be configuredto perform a join operation, combining tuples received from PE4 and PE5.This example operator graph also shows tuples flowing from PE3 to PE7 oncompute node 110C, which itself shows tuples flowing to PE8 and loopingback to PE7. Tuples output from PE8 flow to PE9 on compute node 110D,which in turn outputs tuples to PE10 and further processed by operatorsin a sink processing element 506.

Processing elements 235 (FIG. 2) may be configured to receive or outputtuples in various formats, e.g., the processing elements or streamoperators could exchange data marked up as XML documents. Furthermore,each stream operator 240 within a processing element 235 may beconfigured to carry out any form of data processing functions onreceived tuples, including, for example, writing to database tables orperforming other database operations such as data joins, splits, reads,etc., as well as performing other data analytic functions or operations.

The stream manager 134 of FIG. 1 may be configured to monitor a streamcomputing application running on compute nodes, e.g., compute nodes110A-110D, as well as to change the deployment of an operator graph,e.g., operator graph 132. The stream manager 134 may move processingelements from one compute node 110 to another, for example, to managethe processing loads of the compute nodes 110A-110D in the computinginfrastructure 100. Further, stream manager 134 may control the streamcomputing application by inserting, removing, fusing, un-fusing, orotherwise modifying the processing elements and stream operators (orwhat tuples flow to the processing elements) running on the computenodes 110A-110D.

Because a processing element may be a collection of fused streamoperators, it is equally correct to describe the operator graph as oneor more execution paths between specific stream operators, which mayinclude execution paths to different stream operators within the sameprocessing element. FIG. 5 illustrates execution paths betweenprocessing elements for the sake of clarity.

The operator graph 500 can communicate with the stream manager. Thestream manager manages the flow of tuples within the operator graph 500.The stream manager may further communicate with a fusion manager 145.The fusion manager 145 may be responsible for managing the fusing ofstream operators into a processing element and performing aspects of thepresent disclosure. The fusion manager 145 may communicate multi-fusioninstructions to a compiler.

FIG. 6 illustrates a flowchart of a method 600 for fusing a streamoperator to more than one processing element, according to variousembodiments. In various embodiments, the fusion manager may determinehow to fuse different stream operators based on the readings from thestream operator monitor in the stream manager and transmit instructionsto a compiler to reconfigure an operating graph. The method 600 beginsat operation 606.

In operation 606, the stream operator monitor can monitor a performancemetric for a stream operator of interest. The performance metric is ameasurement that indicates the performance for a specific streamoperator. The performance metric can be relative and based onperformance relative to another stream computing component, e.g., apercent memory usage of the processing element. The performance metriccan also be absolute, e.g., based on a non-relative measurement such astuples/second. Once the stream operator monitor monitors the performancemetric on the stream operator of interest, then the method 600 continuesto operation 608.

In operation 608, the stream manager can determine whether the thresholdfor the performance metric is met. The threshold mirrors the performancemetric. For example, if the performance metric is a percent processingcapacity of the processor on the compute node, then the threshold ismeasured based on a percent processing capacity of the processor on thecompute node. The threshold represents a metric that, if surpassed, willtrigger the fusing of the stream operator of interest on a secondprocessing element that is additional to the first processing element.Thus, if the performance metrics are too high on the first processingelement, then an additional processing element can be used to fuse intothe stream operator of interest.

In various embodiments, the threshold for the performance metric may bemet when the performance metric is greater than or equal to thethreshold. For example, if the performance metric for the streamoperator of interest is 60 tuples/second and the threshold is 60tuples/second, then the performance metric would be met. If theperformance metric is 50 tuples/second, then the performance metricwould not be met. The threshold may be met when the performance metricis less than the threshold. For example, if the threshold for danger forthe stream operator of interest is 90% memory utilization, then thestream operator of interest could shut down. If the performance metricis 60% memory utilization, then the threshold would be met.

If the performance metric is met, then the method 600 continues tooperation 610. If the threshold for the performance metric is not met,then the method 600 continues to operation 606. In various embodiments,if the threshold is met, then the stream manager can provide aninstruction to fuse the stream operator of interest into the secondprocessing element. In various embodiments, the instruction to fuse isprovided to a compiler by the fusion manager.

In operation 610, the fusion manager receives the instruction. Thefusion manager may be a preprocessing manager to determine how aprocessing element gets compiled. In various embodiments, the fusionmanager may also be part of both the stream manager and the compiler.The instruction to fuse is an instruction that, when received by thefusion manager, causes the fusion manager to determine how to fuse thestream operator of interest into the second processing element.According to various embodiments, the fusion manager can receive aninstruction to concurrently fuse, into a second processing element, astream operator of interest that is fused into a first processingelement. In various embodiments, the instruction to fuse can be a fusioninstruction.

The fusion manager can receive the instruction to fuse a stream operatorof interest to a second processing element at run-time via the streammanager or during development time via a developer or user. If theinstruction is received at development time, then the developer of anapplication can initiate the fusion instruction for the fusion managerthat enables the fusing or embedding of the stream operator of interestinto more than one processing element. Thus, the development-timeoptimization can be controlled by the developer. The instruction can bereceived at run-time if triggered by the stream manager in operation 606and operation 608. Once the fusion manager receives the instructions,then the method 600 continues to operation 612.

In operation 612, the fusion manager determines whether multi-fusion isenabled. In various embodiments, the multi-fusion is enabled by virtueof the fusion manager being active to receive the instruction inoperation 610. Operation 612 may be optional or may occur prior tooperation 610. If multi-fusion is not enabled, then the method 600halts. The received instruction in operation 610 is disregarded. Ifmulti-fusion is enabled, then the method 600 continues to operation 614.

In operation 614, the fusion manager determines whether the streamoperator of interest is stateful. The term stateful can describe wheninformation about previous data packets received in a prior streamoperator is stored for some amount of time after those packets/tuples isprocessed. The information from previous data packets can be used toaffect the processing in the current stream operator. A stateful streamoperator includes a stream operator that depends on data from earliertuples than the one being processed. For example, if a first streamoperator performs an aggregating function on the stream of tuples, thenthe aggregator would be stateful because it will use a set of tuples inthe aggregation, i.e., the current tuples plus some number of earliertuples. Another example of a stateful operator includes a countingoperator because the counting operator relies on the number of tuplesthat have already been processed at any point in time. If the fusionmanager determines that the stream operator of interest is stateful,then the method 600 continues to operation 618.

The term stateless can describe when the stream operator will have noinformation about the previous data packets/tuples. For example, if thestream operator is receiving unprocessed data without regard to priorprocesses, then the stream operator is stateless. If the fusion managerdetermines that the stream operator of interest is stateless, then themethod 600 continues to operation 616.

In operation 616, the fusion manager can communicate to the streammanager to fuse the stream operator of interest into the secondprocessing element in response to the stream operator of interest beingstateless. The stream operator of interest can be fused to the secondprocessing element at run-time without compiling the operator graph.Since the stream operator of interest is stateless, then the streamoperator of interest is not dependent on the output of other streamoperators and can be fused into a second processing element. Once thestream operator of interest is fused into the second processing element,then the method 600 continues to operation 620.

In operation 618, the fusion manager can send an instruction to thecompiler to compile a clone of the stream operator of interest into thesecond processing element. The clone of the stream operator of interestis synchronized with the stream operator of interest using a number ofconfigurations described further herein. The clone of the streamoperator of interest shares data with the stream operator of interest ina variety of techniques described further herein. In variousembodiments, the stream of tuples in the operator graph will need to bestopped while the clone is being compiled. Once the clone is created,then the multi-fusion operation is completed and the method 600continues to operation 620.

In operation 620, the fusion manager can receive the data from thestream manager regarding the stream operator of interest and the cloneand determine whether to unfuse the clone from the second processingelement. During the unfuse, the second processing element-clonerelationship can be eliminated by recompilation of the second processingelement. The unfusing operation is discussed further herein.

FIG. 7 illustrates a flowchart of a method 718 of cloning a streamoperator of interest, according to various embodiments. The method 718may correspond to operation 618 from FIG. 6. In method 718, the compilermay receive a signal from the fusion manager to clone the streamoperator of interest and compile the stream operator of interest basedon whether the second processing element shares the same host as thefirst processing element. Method 718 may begin at operation 720.

In operation 720, the compiler receives an instruction to clone thestream operator of interest from the fusion manager. In variousembodiments, the instruction to clone can be a clone instruction or acompile instruction. Once the compiler receives the instruction, thenthe method 718 continues to operation 722. In operation 722, thecompiler can determine whether the second processing element is on thesame compute node as the first processing element. In variousembodiments, the stream manager determines whether the second processingelement is on the same compute node as the first processing element andcommunicates to the fusion manager. The fusion manager determines thatthe compilation is to occur. If the second processing element is on thesame compute node as the first processing element, then the method 718continues to operation 724.

If the second processing element is not on the same compute node as thefirst processing element, then the method 718 continues to operation 726or operation 728. According to various embodiments, the compiler cantake a variety of predefined paths if the second processing element isnot on the same node as the first processing element. The fusion managercan have a preference for either operation 726 or operation 728. Invarious embodiments, the preference is predefined. For example, thefusion manager may prefer that the second processing elements that arenot on the same compute node as the first processing element be compiledusing operation 726. The preference can be based on user preferences.

In various embodiments, both operation 726 and operation 728 can be usedby the compiler if there are more than 2 processing elements with thestream operator of interest. For example, processing elements A, B, andC can contain the stream operator of interest. The stream operator ofinterest can be compiled on processing elements A and B using operation726. The stream operator of interest can be compiled on processingelements B and C, and processing elements C and A using operation 728.

In operation 724, the compiler can compile the clone so that the cloneand the stream operator of interest point to a memory address within thecompute node. For example, if the first processing element and thesecond processing element are on the same host, and the first processingelement has the stream operator of interest and the second processingelement has the clone, then shared memory on the host/compute node canbe used. In this configuration, the compiler of the streamingapplication would generate the proper code to keep the state in theshared memory and lock the shared memory appropriately as the sharedmemory is accessed. In various embodiments, the shared memory could beaccomplished with pointers. Once the clone is compiled, then the method718 halts.

In operation 726, the compiler can compile a state variable into each ofthe stream operator of interest and the clone that is maintained by astream manager. The state variable can be compiled into the streamoperator of interest and the clone but may be managed globally by thestream manager. Thus, when a change occurs, the state variable reflectsthe change. The state variable contains changes of state between thestream operator of interest and the clone. In various embodiments, thestate variable are stored/accessed through runtime utilities andfunctions.

In operation 728, the compiler can compile a communication link betweenthe stream operator of interest and the clone. The communication linkfacilitates communication of state changes between the stream operatorof interest and the clone. In various embodiments, the communicationlink can be compiled by adding send and receive functions to each thestream operator of interest and the clone.

FIG. 8 illustrates a flowchart of a method 820 for determining whetherto unfuse the stream operator of interest or clone from the secondprocessing element. In various situations, the unfusing of the streamoperator of interest can be beneficial in order to streamline theoperator graph. For example, excess processing capacity as a result ofthe multi-fusion operation can be utilized by other parts of theoperator graph. Unfusing can also occur when there is an error producedby a synchronization between a clone and a stream operator of interest.

The method 820 can involve the fusion manager monitoring the streamoperator of interest and the clone for failures or removal instructions.The fusion manager can communicate and arrange the unfusing of thestream operator of interest from the second processing element. Themethod 820 can correspond to operation 620 in FIG. 6. The method 820begins at operation 828.

In operation 828, the fusion manager can monitor the stream operator ofinterest and the clone. In various embodiments, the fusion manager canmonitor the operator graph through data received from the streammanager. The fusion manager can also monitor the operator graph forperformance metrics of any stream operator involved with themulti-fusion operation. The stream manager can also monitor the cloneand stream operator of interest for performance metrics. In variousembodiments, the stream manager can aggregate separate performancemetrics for the stream operator of interest and the clone. The streammanager can further report the performance metrics to the fusion managerwhere the fusion manager makes determinations on whether to unfuse astream operator of interest from the second processing element. Once thestream operator and the clone are monitored, then the method 820continues to operation 830.

In operation 830, the fusion manager determines whether there is afailure on the clone or the stream operator of interest. A failure canbe caused by a number of factors that would ordinarily fail a streamoperator. In various embodiments, the failures can be tied to state. Forexample, receiving a corrupted or malicious tuple from a dependentstream operator can fail the stream operator. In other examples, afailure can be the result of a stream operator causing an exception, atuple received that was not expected, or the compute nodemalfunctioning.

In various embodiments, the stream manager can communicate to the fusionmanager that there is a failure on the stream operator of interest orthe clone. If a failure is determined, then the method 820 continues tooperation 832. If a failure is not determined, then the method 820continues to operation 834.

In operation 834, the fusion manager can monitor whether a removalinstruction is given. In various embodiments, the removal instructioncan be provided by a user based on a determination that multi-fusion ofthe stream operator of interest is no longer necessitated by theperformance metrics. The removal instruction can also be triggeredautomatically by a threshold analysis. For example, the removalinstruction can be triggered by the stream manager if the performancemetric is below a threshold, i.e., too low. If the removal instructionis provided in operation 834, then the method 820 can continue tooperation 836. If the removal instruction is not provided, then thestream manager can continue monitoring for the removal instruction inoperation 828.

In operation 832, the fusion manager can fail both the clone and thestream operator of interest in response to the presence of the failureof either the clone or the stream operator of interest. When both theclone and the stream operator of interest are failed, the fusion managermay immediately halt the processing on both the clone and the streamoperator of interest.

In various embodiments, operation 832 may be considered optionaldepending on a failure policy. A user/developer can set various failurepolicies if the clone or the stream operator of interest fails. In onefailure policy, the user can specify that a failure of either the streamoperator of interest or the clone can fail both the stream operator ofinterest and the clone. The user can also specify that a failure ofeither the stream operator of interest or the clone does not cause thefailure of the other. According to various embodiments, a removalinstruction can be automatically provided in operation 834 if a clone ofthe stream operator of interest or the clone itself fails to function.The failure of the clone or the stream operator of interest can be atrigger to unfuse the stream operator of interest in order to maintainconsistent data. After both the clone and the stream operator ofinterest are failed, then the method 820 continues to operation 836.

In operation 836, the fusion manager can unfuse the stream operator ofinterest from the second processing element. In various embodiments, thefusion manager can receive a removal instruction to remove the streamoperator of interest. If the stream operator of interest is stateful,then the fusion manager provides instructions to the compiler torecompile the second processing element without the clone. If the streamoperator of interest is stateless, then the stream manager candisassociate the stream operator of interest from the second processingelement.

FIG. 9A illustrates a stateless embodiment of an operator graph 900 in amulti-fusion configuration, according to various embodiments. A firstprocessing stream can originate at the source 135, be processed atstream operator OP1 905, stream operator OP2 910, stream operator OPA915, stream operator OP 3 920, and be transmitted to the sink 930. Asecond processing stream can originate at source 137, be processed atOP4 912, OPA 915, and OP5 925, and be transmitted to sink 935. The firstand second processing stream can encounter a bottleneck at OPA 915.Tuples being processed from OP2 910 may wait for tuples to be processedfrom OP4 912 at OPA 915.

The stream operator of interest OPA 915 can receive streams of tuplesfrom OP2 910 and OP4 912. In various embodiments, OPA 915 can bestateless. For example, in a tollbooth system application, OPA 915 cancount the number of cars from a first tollbooth from the firstprocessing stream, e.g., OP2 910, and the number of cars from a secondtollbooth from the second processing stream, e.g., OP4 912.

Assuming that OPA 915 is stateless, then OPA 915 can be fused into bothprocessing element A and processing element B. Processing elementresources can be shared with the stream operator of interest, e.g., OPA915. The multi-fusion operation can benefit the processing speed byallowing more processing bandwidth. For example, OP2 910, OPA 915, andOP3 920 can be served by one processing element without network delays.In various embodiments, the processing streams may be segregated andresult in simultaneous processing of the first stream and the secondstream.

FIG. 9B illustrates a stateful embodiment of an operator graph 940 in amulti-fusion configuration, according to various embodiments. A firstprocessing stream can originate at the source 135, be processed at OP1945, OP2 950, OPA 980, OP3 960, and be transmitted to the sink 965. Asecond processing stream can originate at source 137, be processed atOP4 970, OPA 980, and OP5 985, and be transmitted to sink 990. Assumingthat OPA 980 is stateful, then OPA 980 can be cloned. The first andsecond processing stream can encounter a bottleneck at OPA 980 andresult in a clone of OPA 980, i.e., OPA* 955. For example, if OPA 980aggregated data regarding a toll booth, and OP2 950 determines thenumber of red cars and OP4 determines the number of green cars, then OPA980 can be cloned as OPA* 955.

OPA* 955 can be fused into processing element B while OPA 980 can befused into processing element A. OPA* 955 can synchronize with OPA 980by a variety of methods. For example, OPA* 955 and OPA 980 can bothshare a state variable that is managed by the stream manager. The statevariable can be compiled into processing element B and processingelement B to facilitate synchronization. For example, if OP2 950determines the number of red cars and the OP4 970 determines the numberof green cars, then when aggregating the red and green cars, OPA* 955can receive the number of red cars from OP2 950 and OPA 980 can receivethe number of green cars from OP2 950. OPA 980 and OPA* 955 cansimultaneously share the information via the state variable with OPA980.

In various embodiments, the state variable can be accessed by OPA 980and OPA* 955 after every tuple received by either the stream operator ofinterest or the clone. The state variable can be written to by thestream operator of interest or the clone using a time-based window or acount-based window. For example, the state variable can be written to byOPA 980 every 4 milliseconds. In various embodiments, the operator graph940 can provide for simultaneous writes.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A system, comprising: a compute node that hosts afirst processing element and a second processing element, a streamoperator of interest is fused into the first processing element; amanagement system, the management system having a fusion managerconfigured to: receive an instruction to concurrently fuse, into thesecond processing element, the stream operator of interest, determinewhether the stream operator of interest is stateful, and provide a cloneinstruction, in response to determining the stream operator of interestis stateful; a compiler system separate from the management system, thecompiler system having a compiler configured to: compile, based on theclone instruction, a clone of the stream operator of interest into thesecond processing element so that the clone is synchronized with thestream operator of interest.
 2. The system of claim 1, wherein thefusion manager is configured to: determine whether the first processingelement and the second processing element are assigned to a same computenode; and provide the clone instruction, in response to the firstprocessing element and second processing element being assigned to thesame compute node, to compile the clone so that the clone and the streamoperator of interest point to a memory address within the compute node.3. The system of claim 2, wherein the fusion manager is configured to:provide the clone instruction, in response to the first processingelement and second processing element being assigned to differentcompute nodes, to compile a state variable into each of the streamoperator of interest and the clone that is maintained by a streammanager, wherein the state variable contains changes of state betweenthe stream operator of interest and the clone.
 4. The system of claim 2,wherein the fusion manager is configured to: provide the cloneinstruction, in response to the first processing element and secondprocessing element being assigned to different compute nodes, to compilea communication link between the stream operator of interest and theclone, wherein the communication link facilitates communication of statechanges between the stream operator of interest and the clone.
 5. Thesystem of claim 1, further comprising: a stream manager that isconfigured to: monitor the clone and the stream operator of interest fora failure; and fail both the clone and the stream operator of interestin response to a presence of the failure.
 6. The system of claim 5,wherein the stream manager is configured to: monitor a performancemetric for the stream operator of interest; determine whether athreshold for the performance metric is met; and provide, in response tothe threshold for the performance metric being met, the instruction tofuse the stream operator of interest into the second processing element.7. A computer program product comprising a computer readable storagedevice having a computer readable program stored therein, wherein thecomputer readable program, when executed on a computing device, causesthe computing device to: receive an instruction to concurrently fuse,into a second processing element, a stream operator of interest that isfused into a first processing element; determine whether the streamoperator of interest is stateful; and compile, in response todetermining the stream operator of interest is stateful, a clone of thestream operator of interest into the second processing element so thatthe clone is synchronized with the stream operator of interest.
 8. Thecomputer program product of claim 7, wherein the computer readableprogram further causes the computing device to: monitor a performancemetric for the stream operator of interest; determine whether athreshold for the performance metric is met; and provide, in response tothe threshold for the performance metric being met, the instruction tofuse the stream operator of interest into the second processing element.9. The computer program product of claim 7, wherein the computerreadable program further causes the computing device to: aggregateseparate performance metrics for the stream operator of interest and theclone; and report the performance metrics to a fusion manager.
 10. Thecomputer program product of claim 7, wherein the computer readableprogram causes the computing device to compile the clone by: determiningwhether the first processing element and the second processing elementare assigned to a same compute node; and compiling, in response to thefirst processing element and second processing element being assigned tothe same compute node, the clone so that the clone and the streamoperator of interest point to a memory address within the compute node.11. The computer program product of claim 10, wherein the computerreadable program further causes the computing device to compile theclone by: compiling, in response to the first processing element andsecond processing element being assigned to different compute nodes, astate variable into each of the stream operator of interest and theclone that is maintained by a stream manager, wherein the state variablecontains changes of state between the stream operator of interest andthe clone.